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This  final  report  consists  of  the  following  four  sections: 

1 .  A  summary  of  SRI’s  activities  over  the  duration  of  the  project 

2.  A  report  on  the  Formative  Interaction  and  Engagement  Study  (the  primary  activity  since 
the  submission  of  the  2007  Annual  Report). 

3.  The  2007  Annual  Report 

4.  The  2006  Annual  Report 

Summary  of  SRI  Activities  over  the  duration  of  the  project. 

Throughout  the  project  SRI  participated  in  weekly  meetings  to  advance  the  design  of  the 
ALASKA  system,  as  well  as  suggest  research  directions  for  the  project.  SRI  was  also  intimately 
involved  in  the  project  management  aspects  of  the  task,  helping  the  PI  to  track  budgets  and 
prioritize  activities  based  on  the  available  time  and  funding. 

SRI  completed  the  following  tasks  during  this  project: 

1 .  A  user  test,  conducted  at  SRI  with  SRI  employees,  on  a  preliminary  version  of  the 
ALASKA  software. 

2.  A  series  of  user  tests  conducted  at  SRI  with  pre-calculus  students  from  a  local  community 
college. 

3.  Contributed  to  the  technological  capabilities  of  the  ALASKA  system. 

4.  Created  designs  for  potential  implementations  of  ALASKA  representations. 

5.  Advised  on,  and  helping  to  organize,  a  set  of  AgileViz  workshops  conducted  at  AIED 
and  CSCL. 

6.  Analyzed  video  to  explore  where  the  ALASKA  system  could  be  used  to  improve  teacher 
and  student  performance. 

7.  Advised  on  a  formative  interaction  and  engagement  study. 

8.  Aided  in  the  creation  of  designs  for  the  formative  interaction  and  engagement  study. 

9.  Was  a  key  partner  in  carrying  out  the  formative  interaction  and  engagement  study. 

1 0.  Submission  of  a  report  on  the  formative  interaction  and  engagement  study  (below). 
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Report  on  the  Formative  Interaction  and  Engagement  Study 

By  Phil  Vahey,  SRI  International 
For  the  ALASKA  Project,  PI  6542 


Background: 

At  the  request  of  the  Principle  Investigator,  Eric  Hamilton,  SRI  International  recruited  a  set  of 
four  High  School  Mathematics  teachers  to  participate  in  the  Formative  Interaction  and 
Engagement  Study. 

These  four  teachers  represented  a  cross-section  of  Bay  Area  schools,  and  had  a  variety  of 
technology  backgrounds. 

These  teachers  each  participated  in  two  video  conference  sessions.  Each  session  was  90  minutes 
long,  and  was  held  at  SRI  International,  with  Eric  Hamilton  and  Andy  Hurford  on  the 
videoconference  line.  The  two  sessions  were  held  approximately  one  month  apart,  to  give  the 
ALASKA  team  time  to  react  to  the  teacher  comments  in  preparing  for  the  second  session. 

The  teachers  were  told  to  consider  what  teaching  may  be  like  in  a  few  years,  when  every  student 
has  a  tablet  computer.  They  were  asked  what  types  of  information,  and  what  types  of 
representations  of  this  information,  they  may  find  valuable. 

At  the  beginning  of  the  first  session  the  teachers  told  us  that  the  main  information  they  currently 
get  about  student  understanding  is  found  through  student  hand-raising,  the  questions  students 
ask,  the  answers  students  provide  to  teacher  questions,  and  student  facial  expression.  Of  course, 
while  well-known,  researchers  have  also  reported  these  techniques  to  be  suboptimal,  as  many 
students  become  very  good  at  avoiding  these  types  of  detection  by  teachers.  Such  students  are 
those  that  may  easily  “slip  through  the  cracks”. 

Three  General  Findings 

Three  related  findings  were  striking.  The  first  is  that  the  teachers  were  unanimous  in  their 
concern  about  using  any  technology  if  it  had  the  potential  to  interfere  with  their  face-to-face 
interactions  with  students.  For  instance,  while  lecturing,  they  felt  that  they  would  not  have  time 
to  look  down  at  a  screen  to  view  students’  real-time  questions.  They  also  felt  that  they  would 
usually  not  want  to  employ  such  a  screen  while  students  were  doing  seatwork,  as  walking 
around,  looking  at  student  work,  and  being  physically,  intellectually,  and  emotionally  proximate 
to  the  students  is  perceived  by  teachers  and  students  alike  as  being  high  value.  Taken  alone,  this 
finding  does  not  bode  well  for  the  use  of  technology  the  ALASKA  project  is  investigating. 

The  second  finding,  however,  is  that  all  the  teachers  agreed  that  some  of  the  information  that  can 
only  be  presented  via  technology  would  be  valuable  in  real-time  as  they  are  teaching.  Teachers 
generally  agreed  on  the  utility  of  some  type  of  learning  state  indicator,  thumbnails  of  student 
screens  (assuming  all  students  are  using  tablet  computers),  access  to  full  student  computer 
screens,  and  a  student  “question  bucket”  where  students  could  put  questions  they  have.  There  is 
clearly  tension  between  this  finding  and  finding  1 . 
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The  third  finding  is  a  way  to  resolve  the  tension:  the  teachers  suggested  a  large  projected  screen 
at  the  back  or  side  of  the  room.  While  this  screen  raises  some  issues  around  student  privacy  (e.g. 
will  students  be  willing  to  indicate  confusion  if  all  students  could  glance  over  and  see  the  screen) 
as  well  as  utility  (if  we  show  student  work,  will  all  students  look  over  to  the  screen  to  try  to 
copy),  the  teachers  felt  that  there  were  two  reasons  why  the  screens  would  still  be  valuable.  The 
first  is  that  enough  of  the  information  discussed  would  be  valuable  even  taking  these 
considerations  into  account.  The  second  is  that,  once  the  novelty  wore  off,  students  would  likely 
stop  paying  attention  to  the  screen,  especially  if  it  was  in  the  back  of  the  room  where  it  would  be 
obvious  when  students  look  at  it.  The  teachers  also  felt  that  such  a  screen  may  increase  the  utility 
of  a  teacher  tablet  computer.  The  projected  screen  would  provide  always-on  access  to  some 
amount  of  digital  information,  and  then  in  the  times  that  it  was  not  important  that  they  be 
interacting  with  students  they  could  use  their  tablet  in  a  manner  consistent  with  the  overall 
classroom  flow. 

As  an  enhancement  to  the  original  idea  of  the  projected  screen,  the  teachers  felt  that  such  a 
screen  would  be  particularly  effective  if  it  could  be  controlled  via  some  type  of  wireless 
interface,  like  a  laser-pointer  mouse,  or  a  Wii  controller.  In  this  way  they  could  choose  the 
information  to  see  at  any  given  tine,  without  having  to  look  down  at  another  computer  screen. 

Taken  in  sum,  these  sessions  show  that  teachers  are  willing  to  engage  with  information  displays 
that  can  provide  them  with  more  information  that  they  find  through  the  traditional  measures  of 
student  hand-raising  and  eyebrow-furrowing. 

Detailed  Findings 

The  following  are  summaries  of  the  Question  Sets  that  teacher  completed  while  in  Session  2. 
Question  Set  1: 

We  asked  teachers  to  provide  their  comments  and  thoughts  based  on  the  first  session. 

Q1  and  2:  Teachers  stated  that  they  did  not  engage  in  much  thought  about  the  representations 
between  the  two  sessions.  One  teacher  stated  that  any  display  would  have  to  be  simple  and 
visual,  another  stated  that  using  the  representations  to  foster  student  sharing  would  be  beneficial. 

Q2:  When  asked  to  comment  on  the  specific  displays,  one  teacher  stated  that  the  displays  are 
best  for  sharing  information  with  students,  and  another  stated  that  any  display  would  have  to  be 
very  visual  and  simple  to  use.  While  the  notion  of  simplicity  is  not  a  surprise,  it  may  be  worth 
mentioning  that  this  is  in  slight  tension  with  one  of  the  AgileViz  tenets,  which  holds  that  the  next 
generation  of  teachers,  if  trained  appropriately,  will  be  able  to  use  very  complex  data  displays  in 
real  time,  akin  to  airplane  pilots  use  of  data  (the  team  has  discussed  the  pros  and  cons  of  this 
perspective  in  the  past:  the  author  of  this  document,  Phil  Vahey,  is  not  sure  that  significantly 
increasing  the  cognitive  load  of  display  interpretation  is  a  viable  path  in  the  relatively  near-term, 
as  airplane  pilots  do  not  have  the  added  requirement  of  interacting  with  30  students  in  real-time). 

Q3:  Critical  decision  making  junctures  pointed  out  by  teachers  are: 

•  Questions  students  ask  (3  teachers) 

•  Student  answers  to  teacher  questions  (3  teachers) 
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•  Examining  student  facial  expressions  in  real-time  (1  teacher) 

The  steps  mentioned  by  teachers  that  they  take  based  on  the  outcome  of  these  critical 
junctures  are  (each  only  mentioned  by  1  teacher): 

•  Re-teaching  concepts 

•  Reviewing  procedural  steps 

•  Clarifying 

•  Time  management 

Other  information  teachers  would  fine  useful  included  (each  only  mentioned  by  1  teacher): 

•  Percent  of  students  who  would  want  to  re-cover  material 

•  Percent  of  students  who  understand  the  current  topic 

•  Whether  students  are  ready  to  move  on  to  the  next  topic. 


Question  Set  2: 

We  asked  teachers  to  indicate  their  expected  usefulness  of  the  set  of  tools  presented.  The  results 
follow: 


Tool 

Description 

Usefulness 

Notes 

Resources 

LSI  mapped  to  seating  chart 
Available  objects 

4:  Yes 

0:  No 

Unanimously  agreed  on 
usefulness 

LSI  Screen 

Alpha  list  of  LSIs  and  objects 

Mixed 

Seemed  redundant  with  above 

Thumbnails 

Thumbnail  of  all  student  screens 

4:  Yes 

Unanimous  agreement 

Indiv  screens 

An  individual  student  screen 

4:  Yes 

Unanimous  agreement.  Some 
also  liked  the  ability  to  write 
on  the  student  screen. 

Gradebook 

Shows  all  student  grades 

Mixed 

While  a  gradebook  is  useful, 
felt  that  many  options  exist 

Q  bucket 

Student  Question  bucket,  where 
students  can  input  questions 

4:  Yes 

Agreement  on  usefulness, 
teachers  wanted  control  of 
when  students  could  input  Qs. 
Some  wanted  this  for  only 
after  school.  Some  wouild  use 
this  for  setting  up  student- 
students  interactions. 

Library  Status 

Shows  what  objects  are  being 
used  most  often 

4:  Yes 

Some  thought  they  only 
would  use  the  top  part  of  the 
display,  with  the  frequency 
counts. 

Direct  Help 

Students  input  questions  for  the 
teacher  to  answer  via  IM 

Mixed 

Some  felt  that  students  should 
have  to  ask  questions  in  class, 
others  felt  it  would  help 
students  who  don’t  talk  in 
class.  This  is  one  display  in 
particular  some  thought  would 
be  good  for  a  whole-class 
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display. 

Question  Set  3: 

Teachers  were  asked  to  comment  on  the  displays  used,  if  they  would  use  realistically  use  such  a 
display,  if  a  Teaching  Assistant  would  add  value,  if  the  displays  may  be  useful  outside  of  class, 
and  what  features  or  information  might  enhance  the  display’s  usefulness.  In  this  section  I  will 
only  include  information  not  found  in  the  teacher  answers  to  Questions  Sets  1  and  2. 

Projected  Screen :  this  was  mentioned  by  all  4  teachers.  This  was  perhaps  the  most  important 
original  idea  to  arise  from  the  teacher  sessions.  Additionally,  3  of  the  teachers  mentioned 
wanting  some  kind  of  remote  controlled  mouse  or  handheld  (Andy  H  introduced  the  idea  of  a 
Wii-based  controller). 

Teaching  Assistants:  teachers  stated  that  High  School  teaching  assistant  tend  to  be 
administrative,  and  not  content-experts.  In  fact,  will  often  have  Algebra  students  being  TAs  for 
Calculus  class.  Furthermore,  due  to  privacy  concerns,  TAs  often  do  not  have  access  to  individual 
student  grades.  As  such,  the  TA  can  be  expected  to  make  photocopies,  collect  assignments,  etc., 
but  cannot  do  higher-level  tasks.  They  would  not  be  able  to  make  sense  of,  or  even  legally  see, 
most  of  the  screens  used  in  this  project.  However,  teachers  stated  that  if  they  were  to  have  an 
advanced  TA,  this  would  be  worthwhile. 

Usefulness  outside  of  class:  all  teachers  stated  that  at  least  some  screens  would  be  useful  outside 
of  class.  They  would  use  these  screens  to: 

•  Review  classroom  interactions  (1  teacher) 

•  Better  understand  what  the  students  know  and  what  questions  they  have  (e.g.  Question 
bucket)  (2  teachers) 

•  Review  what  learning  objects  are  being  used  (library)  (2  teachers) 

•  Use  the  gradebook,  but  only  if  it  was  better  than  the  competition,  which  would  be  hard  to 
do  (2  teachers) 
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SRI  2007  Annual  Report  for  the  ALASKA  Project 

PI 6542 

In  2007  SRI  participated  in  weekly  meetings  to  advance  the  design  of  the  ALASKA  system,  as 
well  as  suggest  research  directions  for  the  project.  In  addition,  SRI  completed  several  tasks, 
including  1 .  Creating  designs  for  potential  implementations  of  ALASKA  representations;  2. 
Advising  on,  and  helping  to  organize,  a  set  of  AgileViz  workshops  conducted  at  AIED  and 
CSCL,  3.  Analyzed  video  to  explore  where  the  ALASKA  system  could  be  used  to  improve 
teacher  and  student  performance,  and  4  Advising  on,  and  beginning  implementation  of,  a 
formative  interaction  and  engagement  study.  These  are  described  in  turn. 

1.  Creating  designs  for  potential  implementations  of  ALASKA  representations 

Background 

Key  to  ALASKA  is  the  creation  of  representations  that  teachers  can  employ  in  their  moment-by¬ 
moment  classroom  teaching.  The  SRI  team  created  and  presented  a  set  of  draft  representations 
that  illustrate  how  this  can  be  done  using  an  iPhone-like  interface. 

Representations  and  comments _ 

Seating  chart  display 


Using  the  earlier  ALASKA  idea  of  student  indicators,  students  (or  the  system)  somehow 
indicates  their  current  level  of  understanding  as  green,  yellow  or  red.  Teacher  sees  this  plotted 
over  the  seating  chart.  Imagine  a  grayscale  face  shot  behind  each  of  the  dots  to  remind  the 
teacher  of  each  student. 

The  icons  to  the  far  right  are  used  to  change  the  display.  We  see  that  the  options  available  (so 
far)  include  the  seating  chart  view  (this  view),  a  pictograph/bar  chart  view,  an  alphanumeric  view 
that  shows  students  answer  to  the  last  question,  and  a  current  chats  view  (shows  a  network 
diagram  of  current  IM  sessions). _ 
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The  final  icon  allows  the  teacher  to  control  what  is  on  the  screen  at  the  front  of  the  class.  This,  in 
particular,  is  highly  underspecified. 

By  tapping  a  student,  the  teacher  goes  to  the  student  view  (next) 

Frequency  Plot 


Shows  frequency  of  self-reported  understanding 


Alphanumeric  Responses 


Teacher  asks  “What  is  the  freezing  point  of  water  in  Fahrenheit?”. 

We  made  the  dots  all  one  high-contrast  color:  having  the  dots  be  multi-colored  was  distracting, 
but  getting  rid  of  them  allowed  all  the  numbers  to  run  into  each  other.  We  will  find  some  color 
that  allows  the  number,  and  possibly  the  student  thumbnail,  to  appear. _ 
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Current  chats 


In  this  view,  a  quasi-intelligent  network  diagram  is  shown  (using  standard  network  diagramming 
software).  We  can  choose  how  to  group,  if  icon  sizes  should  indicate  being  a  “hub”,  etc. 

In  this  view  we  may  decide  to  make  it  scrollable,  instead  of  trying  to  fit  all  students  on  one 
screen.  We  will  also  break  the  “grid”  that  we  have  had  until  now. 


Roster 

view 


Srurfam  Uftd 

Qu&k 

Amy  Bayk>r 

B 

Chris  ChGuno 

C 

Enc  Haerukofi 

• 

A~ 

Andy  Hurford 

9 

8* 

Linda  Keyes 

A 

Leo  Lee 

Phil  Vahesy 

8- 

Rotating  the  teacher  device  to  a  portrait  orientation  coverts  the  seating  chart  to  a  list  view. 
Tapping  any  column  header  will  sort  by  that  column. _ 
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Still  haven’t  decided  what  the  icons  below  would  be  used  for. _ 

Individual  Student 


Chris  DiGiano 
Quizzes 

I /1 5/07  A 

vnm 

vnmi  b* 

4/1407  h* 


Understanding  over  time 


Active  Otttti: 

Leo  Lee 
A ndy  H  urford 

Notes: 

Works  well  with  Leo 
Keep  away  from;  PK, 


Tapping  on  an  individual  student  icon  causes  details  on  that  student  to  appear. 


Tapping  any  screen  element  causes  that  element  to  take  over  the  entire  screen  (not  shown).  For 
instance,  tapping  in  the  quizzes  area  will  show  the  entire  gradebook  for  that  student.  Tapping 
active  chats  will  show  the  chat  windows.  Tapping  the  screen  thumbnail  will  zoom  the  screen. 

Individual  Student  Screen 


Chris  DiGiano 


Clicking  this  will  put  the  student  screen  up  on  the  projected  screen.  There  may  be  options,  such 
as  replace  what  is  there,  add  to  what  is  there,  etc. 


2.  AgileViz  workshops 
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The  SRI  team  helped  to  conceptualize,  propose,  and  organize  a  series  of  two  AgileViz 
workshops  that  were  run  by  the  AFA  team.  Both  proposals  were  accepted,  and  the  first  was  held 
at  AIED  in  Southern  California,  the  second  was  held  at  CSCL  at  Purdue. 

3.  Video  analysis 

The  SRI  team  analyzed  video  from  the  TIMSS  study  to  create  a  preliminary  framework  for 
understanding  how  key  representations  in  the  ALASKA  environment  can  be  best  employed  in 
teaching.  This  analysis  has  resulted  in  a  more  nuanced  understanding  of  the  how  different  types 
of  representations  in  the  ALASKA  system  interact  with  different  learning  activities  and  goals. 

4.  Formative  interaction  and  engagement  study 

The  SRI  team  has  been  advising  AFA  on  all  aspects  of  a  formative  interaction  and  engagement 
study.  SRI  has  helped  to  conceptualize  the  study,  design  components  of  the  study  (such  as  the 
representations  discussed  in  task  1),  investigate  teacher  roles  and  recruitment,  and  has  begun 
working  with  AFA  on  the  protocols  to  be  used  in  the  study.  This  study  will  take  place  in  the 
November  to  December  timeframe. 
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In  2006  SRI  participated  in  weekly  meetings  to  advance  the  design  of  the  ALASKA  system,  as 
well  as  suggest  research  directions  for  the  project.  In  addition,  SRI  completed  several  tasks, 
including  1 .  A  user  test  conducted  at  SRI  with  SRI  employees  on  a  preliminary  version  of  the 
ALASKA  software;  2.  A  series  of  user  tests  conducted  at  SRI  with  pre-calculus  students  from  a 
local  community  college,  and  3.  Contributed  to  the  technological  capabilities  of  the  ALASKA 
system.  These  are  described  in  turn. 

1.  User  test  at  SRI  with  SRI  employees 

Background 

Five  SRI  employees  were  in  a  1.5  hour  session  to  use  ALASKA  as  an  aid  in  learning  more  about 
the  SRI  internal  wiki  system.  None  of  these  users  had  ever  used  ALASKA  before,  and  most  had 
some  experience  with  the  wiki. 

Data  Sources 

Observation  notes  (from  Phil) 

Debrief  with  Chris  (the  teacher),  Leo,  and  Mark 
Surveys  completed  by  all  participants 

Overall  impression 

There  were  some  aspects  of  the  system  that  worked  quite  well.  The  teacher  was  able  to  view  the 
work  of  students,  the  students  were  able  to  use  the  chat  system,  and  students  tended  to  use  hybrid 
models  of  communication  (chat  and  offline  talking). 

There  was  a  bug  in  the  parser-buddy  interface  that  severely  limited  the  ability  of  the  buddy  to 
point  students  to  another  student  or  the  teacher. 

While  this  group  has  higher  than  average  technology  experience,  the  overhead  of  so  much 
technology  (screen  sharing,  Alaska  buddy,  multiple  chat  windows,  the  LSI,  etc.)  was  difficult  for 
students  (note  that  students  who  had  the  tablets  liked  them). 

We  may  want  to  reconsider  the  LSI  interface  and  functionality  based  on  our  observations  and 
survey  responses. 

The  set  of  all  technology  available  to  the  teacher  may  be  something  to  consider  moving  forward. 
For  our  situation  we  had  one  computer  being  used  to  project  slides  and  web  pages  to  the  front  of 
the  class,  and  another  for  the  teacher  ALASKA  system.  Different  tech  setups  will  have  very 
different  implications  for  how  the  system  is  used. 

Some  “best  practices”  for  use  of  different  system  elements  are  emerging.  We  may  want  to  keep 
these  in  mind  as  we  conduct  more  research  (that  is,  see  if  they  are  robust  across  different  users) 
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or  we  may  want  to  train  teachers  and  students  on  these  best  practices  in  the  hope  that  these  get 
refined  and  lead  to  more  productive  use  of  the  ALASKA  system. 

SynchronEyes  and  the  overall  teacher  configuration 

This  was  generally  a  success.  The  teacher  was  able  to  use  the  thumbnails  to  get  an  overview  of 
what  students  were  doing,  and  then  was  able  to  click  to  view  students’  screens.  Synchroneyes 
(SE)  was  typically  used  in  combination  with  talking,  and  rarely  with  messaging. 

The  teacher  had  difficulty  moving  between  the  layout  of  students  on  the  SE  screen  and  the  actual 
physical  layout  of  students.  Since  this  class  was  small,  it  was  simple  enough  to  rearrange  the  SE 
screen  to  match  the  physical  layout  of  students,  which  greatly  improved  teacher  use.  For  a  class 
of  30  this  may  be  difficult,  so  in  further  user  tests  we  may  want  to  have  this  set  up  beforehand 
(and  the  final  version  should  allow  the  teacher  to  do  this  beforehand).  Chris  would  also  have 

liked  to  see  the  chat  more  strongly  linked  to  SE. 

♦ 

Due  to  the  small  class  size,  it  was  easy  for  the  teacher  to  click  on  every  student  in  order  to  see 
how  the  class  overall  was  doing. 

The  teacher  had  a  2  computer  configuration.  One  computer  projected  slides  and  websites  to  the 
students,  and  the  other  ran  the  Alaska  system.  This  had  some  benefits,  in  that  Alaska  information 
was  not  being  projected  to  the  students,  and  the  teacher  had  a  clear  separation  of  what 
technology  was  doing  what  (and,  when  the  Alaska  system  was  down  for  a  little  while,  the 
presentation  could  continue).  Unfortunately,  the  teacher  computer  was  not  a  tablet  and  was 
wired,  so  the  teacher  could  not  walk  the  room  with  the  Alaska  system. 

If  we  expect  teachers  to  use  computers  as  part  of  their  regular  teaching,  we  must  pay  attention  to 
the  one  or  two  computer  configurations.  For  instance,  in  a  one-computer  configuration,  can  we 
show  Alaska  on  the  primary  screen,  and  allow  the  secondary  screen  to  show  slides,  etc.?  This 
could  be  similar  to  the  way  that  PowerPoint  currently  works:  when  an  external  monitor/projector 
is  attached,  the  external  monitor  shows  the  slides,  and  the  primary  monitor  shows  a  complex 
control  panel  (in  our  case  it  would  be  the  Alaska  sytem). 

Potential  best  practices: 

•  Have  the  teacher  look  at  the  thumbnail  student  screen  images  to  see  if  any  students  are 
doing  something  wildly  different  than  their  classmates  (or  than  what  is  expected) 

•  Have  the  teacher  click  on  a  few  students  randomly  (or  in  an  informally  stratified  random 
manner)  to  get  a  sense  of  the  overall  status  of  the  class 

•  Engage  students  who  seem  to  be  having  problems,  either  through  chat  or  talk 

LSI 

The  LSI  was  quite  problematic  in  practice. 

The  teacher  did  not  use  the  LSI  as  an  indicator  of  student  state.  It  was  easier  to  look  at  the 
screens.  Furthermore,  the  LSI  was  just  one  thing  too  many — between  teaching  the  class,  using 
SE,  using  chat,  and  occasionally  screen  sharing,  LSI  just  got  dropped. 
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From  the  student  perspective,  the  LSI  was  also  problematic.  It  was  rarely  observed  being  used, 
and  our  survey  results  confirm  that  students  did  not  tend  to  use  the  LSI. 


How  would  you  characterize  the  use  of  the  learning  indicator  as  compared  to  other 
methods  of  letting  the  teacher  know  if  you  needed  more  help  (such  as  raising  your  hand) 
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unsatisfactory  somewhat  satisfactory  very  satisfactory 


Some  quotes  from  the  user  survey  provide  some  insight  into  how  students  though  about  the  LSI. 
Students  were  not  clear  exactly  what  the  purpose  of  the  LSI  was  and  how  the  LSI  was 
represented  to  the  teacher.  In  addition,  they  did  not  take  the  time  to  update  their  own  LSI  while 
engaging  in  the  problem. 

•  I  didn't  see  any  use  in  the  indicator.  The  only  encounter  I  feel  I  had  with  it  was 
seeing  it  on  the  right  side  of  my  buddy  window.  Not  sure  how  it  works  on  the 
teachers  end,  but  maybe  an  alert  should  go  out  to  the  teacher  or  anyone  who  can 
help  out  notifying  them,  and  from  there  whoever  sees  the  alert  can  click  to  accept  it 
and  it  will  go  away  from  the  need  help  alert. 

•  i  really  didn't  get  the  point  of  it.  It  seemed  to  just  take  up  more  space  in  the 
messenger  window.  The  learning  indicator  seems  to  be  inaccurate  of  who  knows 
what—  i  wasn't  really  attentive  to  click  on  the  button  to  let  them  know  my  status  was 
b/c  i  was  busy  doing  the  task. 

•  I  couldn't  see  any  difference  from  being  in  the  'I'm  O.K.'  mode  and  the  'I  need  help’ 
mode.  It  was  more  frustrating  than  useful.  This  option  also  takes  more  screen  real 
estate.  There  should  be  another  way  to  flag  down  the  instructor  (a  button  that 
would  open  the  chat  link  with  the  instructor?)  without  having  to  have  a  status 
showing  in  this  extra  window.  It's  also  not  intuitive  to  have  to  change  the  mode 
once  you're  in  an  O.K.  state. 

In  addition,  the  current  method  of  showing  the  LSI  was  problematic.  The  user  has  to  bring  up  the 
Alaska  buddy,  and  then  choose  “show  LSI”  from  the  “Activities”  menu  (the  specifics  may  not  be 
correct,  but  the  overall  process  is  close).  For  students  new  to  Alaska,  this  is  one  more  thing  that 
they  simply  may  not  do.  Could  the  team  consider  adding  functionality  such  that  the  Alaska 
buddy  can  ping  the  user  upon  sign  in,  and  they  simply  press  an  OK  button  to  initiate  the  LSI? 

In  our  discussion  we  discussed  implications  around  intrinsic  resistance,  or  if  we  can  highlight  the 
importance  of  metacognition  for  students,  and  help  students  develop  more  sophisticated  self¬ 
regulation.  Could  something  like  the  LSI  be  used  once  students  see  that  self-regulation  is 
important?  Part  of  the  Alaska  vision  is  changing  what  people  see  and  do.  The  teacher  should  be 
offloading  cognitive  demands  for  routine  stuff,  and  takes  on  cognitive  demands  for  more 
complex  tasks.  The  student  also  has  to  skillfully  move  in  and  out  of  different  modes  of 
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assistance,  which  requires  more  metacognition.  So,  we  may  want  to  find  technological  and  social 
ways  to  increase  the  use  of  the  LSI  (or  something  very  close  to  the  current  LSI). 

Options  discussed  include:  build  it  into  the  activities,  like  Dylan  Wiliam’s  cups?  Have  Alaska 
buddy  infer  your  LSI  state  (or  just  ping  the  teacher  when  someone  is  particularly  active  asking 
Qs  or  chatting)?  Create  something  that  is  more  like  a  flight  attendant  indicator  button.  May  have 
the  Alaska  buddy  poll  students  on  occasion,  asking  them  how  they  are  doing  in  a  poll.  Also, 
having  the  LSI  appear  on  the  SE  screen  seems  more  appealing,  so  there  is  not  another  screen. 

Alaska  buddy,  MS  Messenger,  and  the  FAQ 

The  FAQ/parser/HAT  interface  had  a  significant  bug  such  that  students  were  rarely  pointed  to 
another  student  when  they  had  a  question  that  the  FAQ  couldn’t  answer.  This  limited  the 
usefulness  of  the  Alaska  buddy.  Nonetheless,  there  were  still  interesting  outcomes  from  this 
study. 

While  most  students  were  observed  talking  to  each  other  instead  of  using  Messenger,  there  were 
times  that  students  did  chat.  In  general  these  chats  seemed  useful,  but  in  a  small  group  of  5 
coworkers,  simply  talking  seemed  more  efficient.  One  comment,  which  is  representative  of  the 
participant  statements  is: 

•  Not  intuitive,  especially  in  a  situation  where  there  are  people  in  the  same  room 
working  on  the  same  issues.  If  the  classroom  needs  to  stay  quiet,  this  tool  would  be 
useful,  but  it's  inhibiting. 

Students  also  initiated  chats  with  the  teacher,  and  these  chats  were  observed  to  be  helpful  on 
occasion.  One  participant  did  point  out  a  weakness  with  chatting  with  the  teacher,  however,  in 
that  one  sends  a  question  out,  and  is  not  sure  of  what  to  expect. 

•  The  lack  of  feedback  I  find  problematic.  The  question  goes  off  into  cyberspace  and  I 
don't  know  if  it  will  ever  be  answered,  if  the  teacher  is  helping  someone  else  and 
how  close  he/she/they  are  to  resolving  her/his/their  problem.  I  don't  even  know 
where  I  am  in  the  queue  or  whether  my  question  is  somewhat  like  someone  else's 
question. 

It  appears  that,  while  the  chat  interface  has  some  potential  in  larger  classrooms  where  cross¬ 
student  talk  is  not  the  norm,  it  was  not  stressed  in  our  user  test. 

While  the  FAQ  rarely  was  able  to  provide  students  with  answers  to  questions,  we  observed 
students  using  the  FAQ,  and  persisting  in  asking  questions  even  when  they  were  becoming 
frustrated  (this  frustration  was  based  on  the  lack  of  their  questions  being  answered,  as  well  as  the 
bug  that  prevented  the  buddy  from  referring  them  to  the  teacher  or  another  student).  A 
representative  statement  shows  that  the  participants  could  see  the  value,  assuming  the  FAQ  was 
well  populated  with  relevant  questions,  and  the  directing  to  other  students  was  productive: 

•  I  like  having  it  direct  to  someone  else  who  thinks  he/she  can  help.  I  also  found  the 

URI  references  useful  (that  is  ,  when  it  had  a  response  rather  than  stating  'I  don't 
have  an  answer  for _ '!) 

So,  while  the  FAQ  and  Alaska  Buddy  were  not  fully  functional  for  this  user  test,  we  see  the 
potential  value  of  these  components. 
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Screen  Sharing 

Screen  sharing,  both  through  SE  and  Messenger,  was  so  complex  as  to  be  difficult  to  use. 

There  were  several  difficulties  that  participants  had  with  the  Messenger  screen  sharing.  These 
include: 

•  Figuring  out  who  was  “driving” 

•  Figuring  out  which  was  the  window  on  the  other  person’s  computer 

•  Understanding  that,  when  a  sharer  covers  part  of  a  shared  window  with  another  window, 
that  part  of  the  window  becomes  black  on  the  sharees  screen. 

Using  the  SE  sharing  capability,  the  teacher  became  confused  as  to  which  screen  was  on  his 
computer,  and  which  was  on  a  student’s  (and  accidentally  launched  an  application  on  a  student 
computer).  In  addition,  the  student  then  received  an  IM,  but  didn’t  know  (because  the  teacher 
was  in  control),  and  the  teacher  wasn’t  paying  attention  to  this.  So,  the  sender  of  the  IM  was  left 
waiting  for  a  response  that  did  not  happen  for  a  very  long  time. 

That  said,  there  are  2  findings  worth  noting  to  the  contrary.  First,  there  was  one  instance  in  which 
screen  sharing  was  valuable:  a  student  had  written  some  text  in  the  wiki  markup  language,  and 
was  not  getting  the  expected  results.  It  was  very  difficult  to  explain  the  problem  to  the  teacher, 
but  by  sharing  the  screen  the  issue  became  obvious.  I  believe  this  was  done  using  the  SE  sharing 
capability,  and  it  is  not  clear  that  the  teacher  needed  full  sharing  capability  to  provide  this  input 
(as  opposed  to  just  seeing  a  picture  of  the  screen).  Second,  while  participants  said  that  screen 
sharing  was  “very  satisfactory”,  three  did  call  is  “somewhat  satisfactory”  and  one  of  the 
participants  said  that  one  of  the  participants  identified  screen  sharing  as  one  of  the  key  benefits  to 
the  system.  Again,  it  is  not  clear  if  a  less  sophisticated  version  (such  as  showing  a  picture  of  the 
screen)  would  suffice. 

How  would  you  characterize  the  ability  to  show  your  screen  to  another  student? 
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Overall,  while  the  ability  to  see  what  is  on  another  students’  screen  is  valuable,  the  current 
implementation  of  screen  sharing  was  very  problematic.  We  may  consider  looking  into  other 
forms  of  information  sharing  that  require  less  cognitive  overhead. 

Screen  real  estate 
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Perhaps  most  surprisingly,  during  observations  there  were  few  complaints  about  the  overall 
screen  real  estate  required  ALASKA,  and  in  the  surveys  participants  were  generally  satisfied 
with  the  size  of  the  Alaska  buddy  window: 


How  appropriate  was  the  size  of  the  aiaska  buddy  window? 
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Other  (minor)  UI  findings 

Participants  were  split  between  minimizing  the  buddy  window  and  leaving  it  in  the  background: 


What  did  you  tend  to  do  with  the  aiaska  buddy  window  while  you  were  working  in 

the  wiki? 
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Of  the  three  students  who  used  tablets,  2  said  that  the  tablets  were  very  accurate  in  recognizing 
their  handwriting,  one  said  somewhat  accurate.  Overall,  this  is  probably  a  higher  rating  than 
expected.  Note,  however,  that  our  chat  didn’t  require  mathematical  symbols,  which  may  give 
different  results. 
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How  accurate  was  the  handwriting  recognition  tool  (if  you  didn't  have  or  didn't  use 
this  tool  choose  not  applicable)? 
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2.  User  test  at  SRI  with  local  community  college  students 
Background 

Five  local  community  college  students  participated  in  a  series  of  3  user  test  sessions,  each  of 
approximately  2  hours  in  length.  Eric  Hamilton,  the  project  PI,  was  a  key  participant  for  these 
sessions  and  found  them  generative  for  thinking  about  the  project.  It  was  thus  decided  that  a 
formal  writeup  was  not  required,  but  instead  SRI  report  on  ideas  for  improvement  based  on  user 
tests  and  related  brainstorming. 

Brokering  help,  and  anonymous  questions 

During  a  help  relationship  brokerage,  allow  a  student  to  forward  a  question  through  the  buddy 
directly  to  another  student 

Provide  a  way  to  ask  a  question  of  another  student  anonymously. 

[note  that  the  two  above  bring  up  a  whole  host  of  questions  around  social  dynamics,  UI,  and 
technology:  when  do  different  people  want  to  be  anonymous,  how  do  they  do  it,  do  people 
behave  differently  based  on  who  is  giving  the  reply  via  the  buddy,  etc. 

Potentially  might  want  an  AQB  (anonymous  question  bucket)  as  a  place  where  folks  can  drop  Qs 
in,  and  get  answers,  all  anon.] 

Teacher  interface  and  synchroneyes 

Allow  the  teacher  to  take  what  is  on  s-eyes  and  display  it  for  all  (either  on  the  projector  or  by 
sending  to  all  students).  Projecting  is  currently  difficult  because  s-eyes  and  the  projector  are  on  2 
different  computers.  Is  there  a  way  to  allow  the  teacher  to  broadcast  a  student  screen  to 
everyone? 

t 

Allow  the  teacher  to  write  on  a  student  screen  from  within  s-eyes  (simultaneous  writing).  This  is 
currently  not  supported  in  s-eyes,  maybe  we  could  convince  them  to  do  it  (since  it  was 
previously  covered  in  a  patent  eric  held,  that  has  since  expired). 

Is  there  a  way  to  zoom  a  student  screen  from  within  s-eyes?  At  one  point  eric  couldn’t  quite  read 
what  was  on  Alvin’s  screen,  and  could  not  zoom  it  to  see  more  clearly. 
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Student  suggested  having  teachers  be  able  to  pair  up  students.  Would  we  want  this  in  technology, 
or  socially? 

The  LSI 

Students  did  not  use  the  LSI.  Recommend  noticing  when  a  computer  is  idle,  and  asking  students 
if  they  are  done  and  can  help,  or  if  they  are  stuck.  This  interacts  with  some  of  the  buddy  ideas, 
below. 

The  FAQ  and  other  resources 

We  have  currently  been  considering  the  FAQ  as  a  resource  for  synchronous  F2F  interactions. 
However,  there  are  other  possibilities,  as  seen  in  the  following  table: 


Physical  Location 

Face  to  face 

At  a  distance 

Timeframe  of 
interaction 

Synchronous 

current  use  of 
buddy 

pre-determined 
study  time 

Asynchronous 

Lab/help  session: 

same  place, 
different  topics 

buddy/FAQ  for 
homework  help 

We  might  want  to  consider  the  other  cells  as  contexts  in  which  the  buddy  and  the  FAQ  can  help 
(initial  possibilities  written  into  the  table). 

Right  now  there  is  only  one  way  to  get  to  the  resources,  which  is  the  FAQ.  The  answers  are  the 
resources,  and  there  should  be  other  ways  to  get  to  the  resources. 

We  may  want  to  consider  what  it  would  mean  to  go  beyond  the  FAQ,  for  instance  with  a  set  of 
resources  that  can  be  used  for  homework.  The  goal  is  already  to  have  the  teacher  drop  applets  in 
and  out  of  work  areas.  This  may  help  us  create/leverage  artifacts  that  are  immediately  accessible 
and  relevant. 

Currently  you  have  to  be  able  to  verbalize  a  question  (and  that  may  mean  translating  from  visual 
thinking  to  textual).  Can  we  invest  more  into  the  buddy  parser  and  have  it  help  students  narrow 
down  what  they  want?  Sort  of  like  a  wizard  or  the  airplane  reservation  system  Chris  mentioned 
before.  Such  a  system  may  sit  on  top  of  the  parser,  and  be  a  large  undertaking. 

We  can  do  a  wizard  of  oz  test  where  there  is  someone  who  helps  narrow  down  questions  and 
give  appropriate  responses.  May  do  this  with  the  WOZ  being  outside  the  classroom,  but  still 
being  a  human.  This  can  be  done  for  multiple  purposes,  including  helping  to  build  up  the  FAQ 
(seeing  what  questions  students  ask),  and  helping  to  better  understand  what  types  of  interactions 
students  find  most  useful  (especially  if  we  WOZ  to  find  out  how  the  buddy  can  help  the  students 
to  better  formulate  a  Q,  and/or  have  the  buddy  proactively  ask  students  if  they  need  help). 

Big  idea:  have  a  TA  be  the  main  ALASKA  user: 
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Now,  if  we  can  use  a  grad  student  in  the  classroom  to  do  this,  it  may  be  even  more  helpful,  and 
require  less  intelligence  in  the  overall  system.  If  this  person  can  manage  the  FAQ,  broker 
relationships,  etc.  It  could  even  be  that  the  grad  students  have  the  teacher  view,  and  the  teacher 
has  a  different  view  that  works  with  the  grad  students.  Note  that  if  there  is  a  huge  class  and 
several  grad  students,  then  each  student  is  assigned  to  a  grad  student.  Big  benefit  is  that  the 
teacher  has  to  be  engaged  on  a  human  level  with  people,  and  doing  that  plus  windows  is  very 
difficult.  Since  this  is  funded  for  research  on  undergrad  classes,  so  it  is  OK  to  do  this. 

Note  that  we  are  killing  2  birds  with  1  stone:  one  purpose  of  this  is  to  create  a  better  FAQ. 
Another  is  to  see  if  we  can  leverage  a  grad  student  in  the  classroom. 

What  would  be  offloaded  from  the  system  to  the  human  (by  using  the  TA  in  the  classroom)? 

1 .  Questions  to  the  FAQ  that  don’t  get  answered  properly 

2.  Human-human  relationships 

3.  An  extra  step  before  the  Q  gets  to  the  teacher  (grad  student  interjects  before  teacher 
would) 

The  FAQ  isn’t  as  effective  as  we  may  have  liked,  so  this  may  (a)  help  us  figure  out  how  to  make 
it  more  effective;  and/or  (b)  help  us  make  the  system  more  effective  keeping  the  FAQ  as  is. 

Perspective:  the  sensory  processing  of  the  windows  is  substantial,  and  a  “shock  to  the  system”  to 
have  so  many  windows  that  give  important  info,  while  also  managing  human  interactions.  So, 
while  the  classroom  of  the  future  requires  “seeing  more”,  that  also  requires  more  processing. 
Having  another  person  would  be  helpful,  and  may  also  help  scaffold  a  teacher  as  they  become 
proficient  (and  many  college  classes  already  have  a  TA).  [potential  concemabout  the  airplane 
pilot  hypothesis:  they  don’t  really  have  to  deal  with  human  interactions  in  the  same  way  a 
teacher  does — does  this  matter?  Human  interactions  take  a  lot  of  processing.] 

We  may  want  to  go  ahead  with  a  non  co-located  task  with  seatwork  time  where  an  ALASKA 
buddy  is  running  (controlled  by  a  human)  and  see  what  kind  of  Qs  come  in  to  the  buddy.  Can 
help  us  gather  “real”  questions.  Can  also  help  with  the  idea  of  the  buddy  always  running. 

Another  idea:  allow  for  better  aggregation,  so  the  teacher  sees  more  than  individual  student 
screens.  Plus,  how  can  an  agent  help  the  teachers,  including  facilitating  getting  resources  to  the 
students.  E.g.  a  count  of  what  questions  students  asking  or  answers  they  are  getting,  better 
polling  of  students,  etc. 

Further  conversation  about  these  topics: 

What  are  our  hypotheses?  At  least  3  big  ideas  (buddy  24/7;  better  FAQs  with  ffee-form 
interaction  (and  a  human  wizard  study);  TA  intermediary),  should  perhaps  pare  them  apart  and 
figure  out  how  to  study  them? 

Should  probably  do  the  human  wizard  to  help  build  out  the  content  area  resources  available.  That 
is,  FAQ  development,  and  building  our  understanding  of  under  what  conditions  would  such  a 
resource  be  useful?  Intermediary  may  also  be  good  about  getting  pilot.  So,  FAQ  +  intermediary, 
can  do  both  with  local  CC  system. 


ALASKA  Project:  SRI  Final  Report 


Page  { PAGE  } 


May  be  tough  to  do  a  good  pilot  with  our  resources,  and  may  have  enough  info  to  serve  as  fodder 
for  a  proposal.  While  we  have  been  looking  at  students  as  a  source  of  data,  but  the  teacher  data 
points  are  probably  even  more  important.  Can  we  have  them  write  down  answers  to  questions, 
etc.? 

The  dialogic  model  of  the  buddy  could  be  good,  where  we  narrow  down  what  the  student  has 
help  with.  This  may  be  a  good  way  to  run  the  problem  solving  exercise. 

We  can  instrument  the  heck  out  of  things,  to  see  when  the  human  wizard  does  interject. 

However,  have  “rules  of  engagement”  to  see  if  we  can  sort  of  model  what  a  computer  would  do 
(Think  of  eliza). 

3.  SRI  contributions  to  the  technical  capabilities  of  the  ALASKA  system 

-  Integrated  FAQ  into  ALASKA  system  and  developed  web  based  front  end. 

-  developed  use  cases.  See  https://wiki.sri.com:  1800/display/alaska/FunctionalRequirements 

-  Held  weekly  technical  status  meetings  with  Leo  and  developed  set  of  milestones 

-  Migrated  old  project  wiki  to  Confluence 

-  Developed  set  of  messaging  scenarios  for  team  to  discuss 
https://wiki.sri.com:  1 800/display/alaska/Messaging+Scenarios 

-  Developed  integration  scenarios  for  team  to  discuss 
https://wiki.sri.com:  1 800/display/alaska/Integration+Scenarios 

-  Developed  proof  of  concept  ALASKA  buddy  using  IM  and  Conversagent 

-  Visited  AFA  calculus  class  and  observed  use  of  tablets  and  SynchronEyes 

-  Attended  project  meeting  at  CU  Boulder 

-  Set  up  technical  infrastructure  for  user  tests 
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